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ABSTPACT 

Questionnaires are prepared by laany people who have 
not had specialized training and experience .in survey techniques. 
This booklet was prepared to assist such nonspecialists in the 
preparation of efficient and useful questionnaires by providing a 
brief, readable guide for the development of ^uestioas for management 
decisions. This discussion is concerned primarily with kinds of 
questions that elicit information about attitudes and opinions. The 
first step in developing a good questionnaire is to examine th-.- 
context in which the data derived from the questionnaire will be 
used. The kinds of decisions to be made, the characteristics of the 
respondents, and the needs of those requesting the information are 
important contextual aspects. Secondly, item formats such as 
open-ended questions, two-way questions, and multiple choice 
questions must be considered along with some of the problems inherent 
in using the various item types^ The items designed and selected for 
inclusion in the questionnaire should be unambiguous, appropriate for 
the respondents, and not misleading. Pretesting the questionnaire 
affords a situation in which ineffective or poorly worded items can 
be discarded or modified. Suggestions are presented for accoaplishicg 
each phase of questionnaire design. (EH) 
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FOI^WORD 



Policy, purchase, and other management decisions are often made 
using information from consumers, students, workers, teachers, or other 
groups with experimental tryout or other, longer-term relevant experience. 
Usually such information comes from a written questionnaire directed to a 

sample of the group. . w * u«j 

Questionnaires are prepared by many people who have not naa 
specialized training and experience in survey techniques. This booklet 
prepared to assist such nonspecialists in the preparation of efficient and 
useful questionnaires by providing a brief, readable gukJe for the develop- 
ment of questions for management decisionB (questions for achievement or 
other tests for individual assessment are npt considered in this booklet). 

This material is based upon work done at HumRRO Division No 4, 
Columbus, Georgia, as part of a research prograir performed under 
contract to the Department of the Army. That work was presented in a 
booklet specifically prepared for Army use, and issued as a HumRRO 
Research Product. A Guide for Developing QuesUonnatre Items, 
Ja..uaryl970. 



Meredith P. Crawford 
President 

Human Resources Research Oi^anization 



A 

5 

1 
1 
I 

I 



< , . ■ CONTENTS 



Page 

INTftODUCTlON ^ 

Section I ASK YOURSELF ^ 

1 . Who needs the infcrmeiion? S 

2. What decision will be made based on your Information? 5 

3. What facts will affect the decision? ^ 

4. Whom are you asking? ^ 

5. What are the consequences Of a wrong answer? 8 

Section II QUESTION TYPES 9 

Open Ended Questions ® 

Two Way Questions , ^ ' 

Multiple Choice Questions ^3 

Section III FRAMING THE QUESTIONS . . . / ^6 

The Initial Development of the Question /: 16 

Pretesting the Questionnaire 26 

Section » A CHECK LIST 28 

A. The Foundation 28 

B. The Question Type ' 28 

C. Writing Questions 



Peveloping 
daesthnnaire Items: 

How To Do It mil 



1 . 

4 



INTROWeiHWI 
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, The purpose of this booklet is to talk about how to ask questions. 
This is not as easy as it sounds. The object is to ask the right question, 
and to ask it in such a way ihnt yon wtti find out what you need 
to know. 

There are a great many pitfalls in asking questions. Not all of them 
witi be considered here. Further, this discussion will deal mainly with the 
kinds of questions that try to get information from a person or try to 
find out how he or shfe feels about something— for example, what was 
learned in a trial of new office equipment, or how a person feels about 
volunteering for military service. There are rules for how best to ask 
this kind of question. Many of these rules will also apply to the kind of 
test that is given after a course of instruction-that is, an achievement 
test-but no attempt will be made to cover all the problems and 
techniques that are important in developing that kind of test. 

Payne' asked tfsample of researchers what they saw as the principal 
problems with research methods. He reported the following results: 

Improperly worded questionnaires 
Faulty interpretation 
Inadequacy of samples 
Improper statistical methods 
Preif^ntation of results without supportmg data 

So, this booklet will concentrate on what seems fc be the largest part 
of the problem-writing gocxi questions. This is not the total solution to 
all research criticisms mentioned, but, as three experts''in every four would 
indicate, it is a good start in the right direction.' 

'Stanley L. Payne. The Art of Axktm Qtu'stiona Princeton University Press, 
Princeton, N. J , 1951. 
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Four S^vtions, tnc*Iuding a filial HtaU'ment summariznig the main 
^working suggesi'ions/* will follow this introduction: 

• Section I: Certain kinds of quf^tions 
you would ask yourself when ibrmu- 
lattng a questionnaire. This is to 
make sure that you know vHiat you 
are asking about. 

• Section 11: A discussio*" of types of 
questions, together w^ i;?ir relative 
advantages and di<iad «ages, Some 
types of questiom' can do certain 
kinds of things- well but do other 
things, fairly poorly. 

• Scvtion HI: A ctisicussion of the 
actual steps in building a question^ 
naire, including some reasonably well 
tested do*s and don^ts. 

• Section IV: A check list that will 
help avoid many of the problems 
involved in writing good questions. 



Seefien I 
ASK YOORSEIF 



Th:' phrase "systems rpproach" is almost a sacred cow these days-if 
anythitiK is done using a "systems approach," it can't be all bad. There « 
some merit to taking a systems approach to asking questions. Asking 
yourself the following five questions will lay the foundation for a far mou 
\:auable questionnaire than you could otherwise produce. 

Who 'leeds the information? 

The reason for asking this question is so straightforward , that it may 
stH?m trivial. However, it is a factor that can be overlooked, and a lot 
depends on it. Organizations have a machinery for transmitting communi- 
cations within Oieir own structun>. The term usually applied m the 
military, for example, is "chain of command." The requirement for the 
information vou are gbing to l olleot may have come from two or three 
levels above your own. ^d something may have been lost m the com- 
munication process. • 

At any rate, there are certain things you must be sure about, and 
knowing who needs the information wUl provide you a source in the event 
you find you cannot answer the remaining four questions. This leads^ to 
the next question. 

. What decision will be made based on your information? 

The answer to this question wUl tell you in part why the information 
is needed and will lay the foundation for the answers the next 
question. This is where the systems approach really starts to make a 
difference. -Depending on what decision is going to be made, some kmds 
of information will make a difference, and other kinds will not. 

Suppose, for example, you are preparing to collect information as a 
part of a- tesi comparing a ne^ item of equipment with an old f t«»daj« 
\^m. The nature of the derision to be made is clcai enough »t be 
either selection of the new equipment or retention of the old. Also the 
test of the new equipment will probably be based on an analysis of the 
requirements or a statement of needs. Exan«nation of such existing 
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documentation—why the new equipment is being considered-^will- usually 
give you a good start for developing your questions. 

Of coui^, there are other reasons in which an explicit statement of 

tJie impending decision may not lead as easily . to a good statemeifit of 
your information needs. However, even in these areas, knowled^ of the 
decision under consideration is an essential key to defining information 
needs-- the next step. 

What facts will affect the decision? ^ 

As you can see, the art of asking good questions may be closely 
related to designing good tests. In order to test new equ4)ment well, it is 
necessary to expose that equipment to the kind of environment and use 
for which it has been designed. This includes use— and sometimes 
abuse— by the type of individual who will use it operatio'raily. 

When yon need the information for something less straightfcrward 
than an equipment test, the task of finding out what facts will have* a 
significbnt bearing on the problem is not always easy and will sometimes 
be downri^t difficult. Sometimes the decision maker himself may be 
unsure of the ft.J range of information that will be needed to make a 
good decision, in this case, learning what kind of facts will help impart of 
the probleii-. If this" happens, you will find it useful to consult with 
colleagues, asking, in essence, "What kinds of things would you want to 
know if you had this decision to make?" It will help to make a list of 
elements of information ;hat you feel will be needed as you go. If you 
have time, you may want to have this list reviewed before you start 
developing questions. 

Clearly, the real question i&, "What information will maka a 
significant difference to the decision maker?" If you collect unnecessuy 
information, or fail to get the needed information, your time will iiave 
been put to no good use^ at the very least. A considerably more serious 
possibility is lhat the decision may get made on the basis of irrelevant 
'information. This leads to our fourth question. 

4. Wl!^m'are you asking? 

To get good information, not only must you ask a good question, 
but you also must ask it of someoSie who has the answer^ This ties in 
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with the design of tests, which exposes proper test ^subjects to the 
proper conditions so that they will then be able' to t«ll you the kind of 
information you want to know. In fad, one basic objective of a good 
equipment test is to qualify the jjeople who will take the test to be 
able to answer the questions that need to be asked. It is also important 
to^ sure ^hat the appropriate kind of test subject is used. 

• 

SomeUmes it is necessary to get information without the benefit of a 
carefully designed test. When this is so, it is especiaUy irilportant tft be 
sure you are asking the right pereon for the information, m terms of 
whether he is qualified to answer and whether Jie has the information you 
need, if qualified. This may even .require you, in some cases, actually to 
seek out an expert to be certain of getting good answers. 

It does not requir? much knowledge of how to ask questions to 
know that a question about billing ibr oil should not be asked of a 
bookkeeper for the oil company. However, sometimes the problem is 
more subtle. Consider the'Tollowirig question: 




I Example 1 . 

Was the TOE of your unit suitable for ttie 
missions assigned? 



This question was asked of noncommissioned officers in Vietnam. While it • 
appeared to be a r)erfectly reasonable question to the officer who wrote it, 
he had not stopped to think thatr NCOs generally have not been trained to 
think in terms of how. Tables of CrganizaUon and Equipment.are designed 
to proHuce missiion effectiveness. It is quite possible that many noncom- 
missioned officers answered this question on the basis of whether they 
thought the missions assigned to them were reasonable missions, which is 
quite a different empha.sis than that requested i|i the question. 

A very important consideration is' the fi'amc oi reference the test 
subject will be using in- answering the question. Frame of reference was 
the Mumbling block in the example just given, in which the NCOs frame 
of reference is simply different from that of the officer who worded the 
question. Frame of referen<:e will be discussed in greater length later. 
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Another topic that will be t-on^idere.* is whether 'the test subject can 
understand ^what you want in a question. Ihe Army has an expression that 
. , is time-wom but to the point: KISS. This stands for "Keep it simple 
stupid!" We will return to this topic later, too. - ' 

5. What are the G9nsequences of ^^ong answer? 

While this basically is an administrative question, it has an important 
beari'n? on questionnaire and test design. Qearly, if it makes little differ- 
ence whicli one of two alternatives is chosen, it makes little difference 
whether the information is collected. On the other hand, if there is a 
chance that an organization could save millions of dollars throu^ the use 
of a more effective block of training, or mi|^t spend millions of dollars 
for .1 new piece of equipment that is no better than the oW, it is necessary 
* . to design tests very well, and ask the right questions with great can*. 

Also, in genera!, the more ixiiportant the dteision or the more 
precise the needed information, the, greater, the number of test subjects 
needed in order *to provide information of sufficiently high reliability* 
AgaiA, this pertains more to test design and the statistical analysis of 
th^ data collected, the important point for the present discussion is 
^ simpily that, whon you ask questions, you must have a feel for how 
important it is ^at the answers are right. 
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Seefloyi II 
QUBTIOH TYKS 



Only three mctfor types of questions wttl be considered here. There 
are many variants in handling questions, and it wUl almost always be 
possible for you to ^et the information you need with one or mo^e of 
these three types. The types are: 

• Open-ended \^ 

• Two-way 

• Multiple-choice 

In deciding which type you ought to use, you wUl need to consider 
the following factors: 

(1) How much you know about the range of possible answers. 

(2) How much time your respondents have, or are wUling to 
giveyou. 

(3) The number of subjects needed to get satisfartory reliability, 
and the resulting requirements for data analysis. 

Open-Ended Questions 

An example of an open-ended question is shown below. 

I Example 2 

What do you think of the short income 
tax form? 



One advantage of the open^noed question is immediately apparent. 
Ir is very easy to .dsk. This is useful when the question wnter does not 
know or is not certain about the entire range of possible alternative 
a»«wers. By using this type of question and by asking enough respondents, 
you can usually discover most of the possibilities. 



However^ open-ended questions have some disadvantages: 

• It usually will take a good deal more 
time for you to get answers, if y /U 
expect the answers to be written. In 
fact, depending on the motivation of 
your respondent, he or she may not 
give a full answer, or any answer at 
all, after seeing how much work it is 
going to be. 

• Open^nded questions demand a lot 
of analysis time. For a question askor 
on a crash project, the only thing V 
that couki be worse than failing to 
get enough informatica would be 
gettmg so mu.-h that he is buried by 
it. Open«ended questions, answered 
by motivated respondents, can over- 
whelm data analysts. They usually 
cannot be handled by machine 
analysis methods (e.g., computer data 
proces^ng methods) without lengthy 
preliminary steps. If your test , 
retjuires a fairly lai^e number of sub- 
jects, or a feiriy large munber of 
questions, the data analysis problem 
can grew into a major project unless 
some other form of question is used. 

• Responses to an cpen-ended question 
usually must be analysed by some- 
one who has substantial knowledge 
about the question*s content, as 
opposed to a statistical clerk, for ^ 
example. This w&y result in imposing 
another work load on someone who 

is already hard pressed for time. 

• Open-ended questions m^ be easier 
to misinterpret, because the re£q[>ond- 
ent does not hai^e a set of choices 
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avaUable which might in themselves 
guide him toward the proper frame 
of reference. For instance, the ques- ^ 
t>on in Example 3 is given by Payne: . 



I Example 3 

Why do you buy the 50-cent motor oil 
in stead of the 70-cent oil ? 

A fairly obvious reason for buying 
50-cent ott instead of 70-cent oU is 
that it is cheaper. However, many 
lespondents may feel this is so 
obvious that the person asking the 
question wants a different kind of 
answer. Thus, the question may fail 
to produce valid responses. 



In summary, the open-ended question is easy to a«k but it h^ 
several disadvantages; in particular, it produces data *^at are time- 
consuming to analyze. Sometimes, a good procedwe ^ ^ }^ ^ 
open^nded question with a small number of respondents in o^er to 
fmd out what the mnge of alternatives is. Armed with this knowledge 
you may then be able to construct good multiple^jhoice questaons that 
will be faster to administer and easier to analyze. 



Two-Way Questions 

The two-way question is just what it says-^ question that permits 
choosing one of two alternatives. Example 4 is a two-way question. 



I Example 4 

VWhich do yqu prefer, the ABC typewriter or 
the XYZ typewriter? (Check one) 



ABC typewriter 
XYZ typewriter 
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An advantage is immediately apparent. Your respondent can merely 
check one answer and go on to the next question. It is a relatively easy 
type of question to develop and permits rapid analysis. 

However^ it also has disadvantaf^es. First, two alternatives mi^t not 
be enough for some types of questions. . Second, there may be a tendency 
for your respondents to choose an answer based on what is known as a 
respctise set. That is, some people tend to choose the first answer they 
see, while others may choose the last answer they see. A way of avokiing 
this problem with the two-way question is to print jialf the questionnaires 
with the question in the form shown in Example 4, and the other half 
with the order of mention of the two typewritcf)^ reversed. Sometimes this 
will make a difference and sometimes not, ^depending to a substantial 
extent on the number of words in the altematiWs. However, it is a good 
precaution to take even with diort aHematives like those used in 
Example 4 on the preceding page. 

Another common fault of the two-way question is the presentation 
of alternatively that overlap (not mutually exclusive). Example 5 has 
overlapping alternatives. 




While this question has more than one problem, it is apparent that it 
is quite possible that the ABC typewriter is both a good typewriter and 
could be improved. Almost anything can be improved. Thus, the second 
choice ought to be extremely popular if your respondents think carefully 
about it, without regard to the merits of the typewriter as such. 

In summary, for some purposes the two-way question may be an 
improvement over the open-ended question, in that it provkies for faster 
and more economical analysis of data^ which means that data can be 
obtained from more respondents. However, it does require a bit more care 
in its development. Further, it is also subject to some of the faults of 
multiple-choice questions, as well as some peculiar ones of its own. 
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Multiple-Choice Questions 

This type of question has the advantage of being easUy scored. Which 
means that data analysis is a relatively mexpensive process .•eqmrmg no 
special content expertise. It also requires considerably less time per 
respondent answer than the open-ended question. Further, it does not 
require respondents to work so hard to give you the mformation you 
need, which means they are more likely to try to do what you want. 

Perhaps an even more significant advantage is that it puts dl pereons 
on the same footing when answering. That is. each person will be able to 
consider the same range of alternatives when choosing his answer. On the 
other hand, this may be a major disadvantage for the question maker, 
because it requires him to know aU the significant possible alternatives at 
the time he formulates the question. 

One of the big problems with multiple-choice questions is being 
certain that the answer you receive to your question is a real answer. This 
statement may sound ridiculous, but the fact of the matter is that a wry 
deliberate and well thought-out response on a paper-nnd-penal multtple- 
choice test looks very similar to one that has received no thought at aU. 
That is, both have check marks, circles, blacked^ut spaces, etc. How- 
ever, there are a few things thay you can do to make it more Ukely 
that the answers you get are well thought out. 

As was noted in discussing the two-way question, people have a 
tendency when choosing answCTs to be influenced by "response set. 
With multiple-choice questions, for example, they tend to choose 
answers from the middle of the list if the list consists of numbers, and 
from either the top or the bottom of the list if the alternatives are 
fairly lengthy expressions of ideas. A way to combat this for numbers is 
to use different forms .of the ouestion with the alternatives m a 
different order, for different pans of your sample.' ' For alternatives 
presenting ideas rather than numbers, this same approach can be usea. 
in addition to keeping the number of words in each alternative smaU. 

Another kind of error Jie inclusion of an alternative that is 
obviously very desirable. Example 6 contains an example of a sociaUy 
undesirable response. 

•Where the number of respondents and/or questions b large, the added burden 
during data reduction can often be handled by computer file editing routmes which 
"rearrange" the alternatives for analysis. 
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f Exampla 6 



What percentage of the time were the 
enemy targets: 

a. Exactly located and 

fired at 

\ b. Closely located and 

fired at 



\ c. Location unknown but 
\ fired at hoping to suppress 
\the enemy 



% 



Attemattve if the soldier respondent thinks carefUUy about it, 
might be s^n as implying a lower degree of skill in locating the enemy 
than might be desirable. In thinking about the item, he might see that 
he could check b^ instead, and still live with his conscience. Admittedly, 
he did not see the enemy but he can be fairly well convinced that ihe 
enemy was "closely located '*~that is, he was not firing purely random 
shots as alternative c^ might imply. Thus, the pure social desirability of 
the alternatives ought to result in higher percentages reported for and 
lower percentages fore. 

Example? shows another form of this same problem. If a motor^ 
cyclist IS asked this question, he almost certainly will choose protection as 
a helmet's most important characteristic. 

Exampitt 7 



What do you consider the most important 
characteristic of a good motorcycle helmet? 

a. Comfort 

b. Stability 

c. Attractive appearance 

d. Protection 

e. WeiQht 



However, if the test helmet must meet the same safety require* 
ments as the standard helmet, the fact that protection is dwired by the 
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cyclist ts irrelevant. The test helmet would be failed anyway in other 
'testing if its protection capabilities were not adequate. Asking the cyclist 
this question, then, adds no new information. Indeed, it obscures 
possibly important additional factpts thatr-wottW have -appeared as 
second >:hoic^. 

This gets back to the requirement for a-systems approach to the 
asking of questions, and tO the "So what?" you must ask about your own 
questions. In this particular case, the answer to "So what?" would be 
"Nothing." Either this question should not be asked, or it should be asked 
in a way that will give information about what the motorcyclist considers 
of next importance after protection. This can be done either by leaving 
out "protection," or by asking respondents to nfnk the alternatives. 

In summary, multiple-choice tests are easy to administer and fast to 
score. They are especially useful for dealing with a large number of test 
subjects. However, the questions and their alternatives must be worded 
with very great care, or the information obtained ftom them may not be 
vaTid A particular requirement for multiple-choice items is that the ques- 
tion writer must know the fuU range of significant alternatives. However, 
he can help himself in that respect by asking a few subjects to respond to 
open-ended questions in advance, and can give himself some msurance by 
providing an open-ended alternative which can be handled m the data 
analysis through special processing, as is shown in Example 8. 

Example 8 



How much education have you had? 

a. Did not finish high school. 

b. FiniSiied high school, but 
nocollega.- — 

c. Some college, but did not 
graduate. — 

d. Graduated from college. 

but nothing beyond. — 

e. Some graduate work. 

f. Other (please list) 
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The general considffations discussed in the preceding two sections 
should help in giving you an initial idea about the extent of the questions 
you want to ask and the way you want to ask them. The question of 
what infonnation is needed should have been answered by the systems 
examination of the requirement. This should have provided a compre* 
hensive list of all the points on which information is required for the 
decision that is going to be made. The brief consideration of question 
types should have given ypu an idea of how you want to ask 
your questions. 

From this point on, the development of the questionnaire consists of 
initial framing and review of questions, and pretesting. 



The initial Development of the Question 

\jot us assume that you have selected a particular is^e or element of 
information about which you now are going to trmte a (Question. Here are 
some cautions that can help you in writing questions. 

1. Don't assume your respondent will know what you are ta/feing abouU 
just because you do. 

Example 9, taken from Payne, illustrates this point..^; 




The odds are that a fairly substantial number of people would not 
be able to define these two question types for you. Nevertheless, if you 
ask them the question in Example 9, they will be happy to choose. The 
;)Otnt is that people will not voluntaer their ignorance of somethings 
though they may admit it If you ask them. 
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However, this ciiution goes beyond ignorance of an issue. Another 
problem is that the specialist wording the question may simply have an 
unusual command of his own language: Scientific and bureaucratic jargon 
has often been criticized. There are other kinds of jargon, too The 
quesUon asker has a responsibility to make himself understood. 

In case you need to screen out individuals who do not have an 
adequate basis for providing the information you need, one way to do 
this is to include one or two pure information questions; then discard 
questionnaire returns frc.a respondents who cannot answer the informa- 
tion questions correctly. 

2. Don't leave yourself wondering: What did fcf mean by that answer? 

Sometimes, a question can be worded sp tha). it is impossible to 
know vhat wa«; meant by a given answer. Example 10 Ulustrates this. 




If the student answers Yes, what wouW you say he meant? It is 
unlikely that you would write a question with such obviously ambiguous 
response alternative, but it does illustrate the problem, 

» 

3. Don't ask people to go against their basic inclinations. 

This is a very vague statement and it applies to many things. An 
example is that people generally are reluctant to criticize, though they 
enjoy giving praise. (There are some exceptions to this rule, but it is 
generally true.) Thus, a question that allows a respondent to avoid being 
critical will bias answers. 

SimUarly, a question that leads him to criticize, especially on funda- 
mental issues, will bias responses becau«e-1ie will not wish to do so. 
Example 11 illustrates^ this. 




This question asks the respondent either to cxiticize the action of his 
own group or to avoid being critical. Some hardy souls mi^t answer No« 
if they have an important point to make« feeling that the explanation can 
put them back on good footing. However^ substantial number of others 
will wash their hands of the whole affair and answer Yes» no matter how 
they feel on the subject. 



4. Don't ''leadr 

You have probably heard, several times, the expression^ '^People are 

no d good!** The problem with this is that it simply is not true. 

People generally are reasonably cooperative and like to help. If they can 
figure out what you want, they wOl try to give it to'^you. Example 12 
contains two illustrations of leering. 
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The impression one would be likely to receive from analysis of the 
answers is that students are generally highly motivated and appear to 
take pretty good notes during the instructor*s lectures. (These questions 
also suffer from allowing respondents to avoid criticizing the situation*, 
and of allowing them to make socially desirable answers as well. This is 
a problem that will be discussed later.) ' 

t 

\ 

5. Don't confu^ or lo^ the respondent. 

The language you use in framing your question will have a lot to do 
not only with the quality of the information your respondent gives you, 
but also with how he feels about giving it. One of the easiest ways to 
create a problem is to use a confusing question such as that given in 
Kxample 13 (taken from Payne). 



This illustration might mean anything. However, it is not necessary 
to use double negatives to confuse someone. It can be done more 
simply, merely by using words he cannot understand, as shown in 
Example 14. 






Examine 14 



What countermeasures should you take to * 
preclude having an automobile accident 
from a blowout? 



a. With a heavy sedan 



b. With a lighter compact 



Of course^ ibis question has additional problems other than the fact 
that '*countermeasures*' and "preclude'* are words that might not be 
understood by everyone. One such problem is that a respondent who 
knows all the words stiU may not know whether your emphasis is on 
preventing bhwouts or preventing accidents after experiencing blowouts. 
There is ' a difference, and your data will be a mixture of both 
interpretations if the question is not clear. 



Example 15 ^ows how a respondent canbe asked for information he 
may not be able to give. 



Emmple 15 



Vjlhat special techniques did your supervisor 
teach you to use with the foltowtng types 
of office equipment? (List only actions not 
now covered by our company operating 
manuals.) 

a. Fiexowrlters ' 

b. Duplicating machines 

c. Eie^rostdtic cppters 



This particular source of trouble* as far as this point is concerned* 
comes from the section in parentheses. While the intention of the question 
writer is fairly striifi^tforward* he has virtually eliminated most respondents 
because most* unlike the author of the question* are probably not familiar 
enou^ with company operating manuals to know what is not included. 

Of course* two other problems should be menticmed about this 
particular item* First* the word '^actions** is ambiguous. It has two 
meanings* and it is difficult to know iniiich one is correct in relation to 
the ^ntence preceding it. Second* this is an open-«nded question with a 
vengeance. Given a motivated re^ondent in a hi^ly innovathre oiganisa- 
Uon* it is possible that an answer to this sini^e question could take hours. 
It clearly is a question that is more convenient for the writer-^t the 
expense of the respondent. 

Finally* Example 16 shows that several different methods, of con* 
fusing the respondent can be combined in the same question. The first 
problem is in the wording Of the initial question. 
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Example 16 



When engagiito enemy targets, seen or suspected, at rartfies 
from 0 to 50 meters and under extreme time pressure, what 
type of fire would you use? 



(203) 
(20&) 



(200) 



(178) 



a. Semiautomatic. 



automatic . 



b. Carefully aimed quickly aimed using the 

sights : pointing type not using sights fired 

from the shoulder ; pointing type underarm \ 

c. Single rounds 2-3 round bursts ; 3-S round 

bursts ^;510 round bursts ; 20 round burst 

"hose effect ^^^^^ . 

d. Fire at single targets^ 
personnel ; fire at the area 



; fire at groups of 



(186) e. Shoot first then take cover. 



take cover then 



shoot. 



J wait for orders. 



It is doubtful thfit th> average infantry soldier, when leading this, 
would really have grasped al the conditions stated. , 

This confusion might be reduced by rewording, perhaps in this way: 

What type of fire would you use when engaging wiy close targets under 
great time pressure? 

♦ 

The plan here is to make the question more understandable, and aSso to 
break the very complex initial quesUon into parts that can be handled 
more easily. 

A second fault with Example 16 is that it Is not reaUy clear that the 
soldier is to respond to each Une^ In this example* the numbers at the left 
of each line show how many answers were given to all the possible choices 
on that line when this question wto actually used. It is clear that while 
almost everyone grasped the idea that he should answer more than one 
time, not everyone answered the same number of times. However, it is not 
certain whether some answered too niany or some too few, or both. 
Difficulties like this can be avoided by giving the respondent explicit 
directions, and by making his job more simple than this question did. 
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6.Aon*t make your queations too long. 



I 

This, and the preceding po\n{^ are similar considerations when wfitii^ 
questions. .The more words a 'question has and the bigger the ^ words. are» 
the more' confusing the question is likely to be. Example 17 illustrates 
this problem 



I Example 17 i 

In the higfjiy specialised counterTnsurgency 
environment represented by the bastoaliy 
intrfrnecine affair in Vietnam, what would you 
>say should represent the ^asic essence of our 
rattonai^ for continuation of our involvement? 

a. Prolongation of attrition 
of enemy forces, in order 
to redDce the level of threat 
to South Vietnam. - - 



Orderly transfer of military 
responsibility to the host 
country, in order to pro- 
duce stabilized contpetency 
to deal mth any future 
internal disturbances. 



Several things are wrong with this question. However, it is unlikely that 
the average respondent would be bothered by^ them^. Thj^ massive words by 
themselves very probably would convince a subject to stop readingCRt an early 
point. He qr she might then choose an alternative by flipping an unbiased 
coin. An even more attractive choice might be just to go to the next item. 

Worst of all, the respondent- probably will choose one alternative, but. 
not in an unbiased manner, so that your item not only will fail to obtain 
valid information, but ivUt obtain invalid information. 

The following four "word** rules, adopted from Payne, provide a very 
useful guidance: 

• Does the y^rord mean what you 
intended? 

. - 2,7 
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• Does the word have any other 
meanings? 

• if so, does the context make the 
intended meaning clear? 

Is a simpler word or phrase suggested 
in the dictionary? 



Perhaps a fifth word rule might be: 



Can you get along without it? 



The point of this caution about choosing words is that it is very easy 
to get incorrect information on a multiple-choice test by giving your 
respondent a choice he cannot understand. Most people will cooperate. 
They will choose anyway. . 

When actually writing questions, you' should carefully screen each 
question for words that might be unfamiliar to . le respondents you intend 
to use. Pretesting, which • will be discussed later, can also provide an 
opportunity to look for difficult words. A good technique for pretesting is 
to have the respondent read the question aloud to you and then tell you 
what it means. Any difficulties at all in the pretest should cause danger 
signals to fly. 

T. Don't use ''Atl-American"" or ''giveaivay'" words. 

This caution is related to the one juft discussed. Kxample 18 shows 
the use of two *'dead giveaway'* wordi which would lead the careful 
thinker to respond in the negative, and others, thinking less carefully, to 
respond in the positive. 

I Example 18 

Do you feel that your department did its 
best in atl aspects of its work over the past 
six months? 
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One wonders if any group of employees can do their actual best, except 
very rarely. The word all makes this an even mate difficult question to 
ar wer positively. You probably have heard the expression: All stateinents 
containing "all" are false, including this one. 

The effect of a question like that shown in Example 18 is that it 
ma> separate one kind of thinker from another, without regard to the 
kind of information you are trying to get. 



8. Don "t load your questions. 

Loading questions is a good deal like loading dice. Either way, you 
can be i>retty sure in advance what kind of result you are going to get. 
There are many different way* of loading questions. Of those given by 
Payne, the following appear to be the most likely pitfalls for pres- 
ent purposes. 

a. -Tl\e use of stereotypes. If motherhood and sin were on a ballot, 
most people would vob W^oti^tn^rtSDd^^and against sin. Example 19 
shows a motheiiiood type 




Did you clean your weapon regularly in 
Vietnam? 



As you can see, fJjis kind of question is not only loaded; it's very 
much like leading, as illilstrated in Example 12. 

b. Recourse to the status quo . People have a substantial tendency to 
keep things the way they" are, unless something is really wrong. 
Example 20 shows a question that shouki have been worded in-a dtfferent 
way to avoki this problem. 
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Answers to this question should be suspect merely on the basis of the 
tendency people have to keep things the way they are, 

c. Unbalanced alternatives. Example 21 shows loading of a different 
type. In the.iTirst part of this example, analysis of the available alternatives 
leaves the impression that the writer of the question thinks that not aU 
libraries sK^uld have a complete reference room. Analysis of the alterna- 
tives in the second part of the example leads to the certain conclusion 
tnat the writer of the question believes there should be a copying machine 
in the main library. 

I Example 21 

Should ev«»-y branch of the city public library 
have a reference room with a itOfM. identical 
-to the main library? 



Yes 

No 

if No. list those 
branches that should. 



How many of the city's libraries should have 
some kind of copying machine? 

a. Main library only, . . 

b. Main library plus bianches 

serving the central city. ^ 

c. Main library and several 

branches serving the suburbs. : 

d. Main library plus all branches. 
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This leads to the observation that any bias on the part of the 
question writer himself just migkt be one source of loading. It turns out 
that this is very oftem true, and most of the time is not recc^ni^ed by the 
writer at all. So a good check here is to ask yourself what you think, what 
someone who disagrees with you would think, and whether your alterna^ 
lives would give him a good chance to present his views. 



Pretestini^ the Questionnaire 

Even the most careful s^creening will not find every problem with all 
questions. Pretesting thus is very important, to find those overlooked 
problems that would reduce the validity of the information you will get. 
However, mst any pretest will not do. You must know how to pretest 
the items, and what to look for. 

Perhaps of first importance, the pretest respondents should be 
representative of your eventual target audience. That is, if a test is going 
to be run using high school freshmen, you would not want to pretest 
your questionnaire with a sample of univeraf^ graduate students. Their 
ability levels would be grossly different. Similarly, if your eventual 
sample is to be apprentices in one of the trades, you would not want 
to pretest with experienced journeymen whose experience levels would 
be grossly different, 

A second point is that the pretest is more useful if it is conducted by 
scnieone who knows the subject area. If the question writer himself^ for 
example, conducts the pretest, using pretest respondents one at a time, he 
can often obtain useful information by asking for the question to be read 
and then explaine^. Misunderstandings become very apparent throu^ 
this process. 

Another useful technique is to ask the respondent to explain what he 
means by a choice, or to give his reason for the choice, even though these 
questions will be not asked in the main administration of the ques^on* 
nairr. These questions will frequently reveal incorrect assumptions and 
iu>ss!bte rationales that the question writer never dreamed possible. 

During pretesting cf the questionnaire, a high proportion of 
respondents giving no answer or a ''don*t know** response should raise 
danger flags. However, especially for a multiple-choice question, a low 
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number of **don*t know" tesponses does not guarantee that the questions 
are good. By recourse to the methods mentioned in the preceding para- 
graph, the protester can come closer to guaranteeing that the questions 
are good. 

PretesUng may seem a waste of time, especiaUy when the author may 
have asked several people in his own office to critique the questions, or 
perhaps even has asked a questionnaire specialist to critique it. However, it 
will usually be an investment that is weU worthwhile. It is cmcml if the 
decision that wUl result from the information is important. 
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In the preceding Sections, we have talked first about the kind of 
information you need to establish as a foundation before you start Mniting 
questions, about the types questions among which you may choose, 
and abc>ut the kinds of problems that may confront you as you develop 
the questions for your questionnaire. The following check list may be of 
use to you both as a review and as a way of refreshing your memory as 
you actually work through the steps of developing a questionnaire 
instrument. 

A. The Foundation 

1. Find out who needs the information. 

2. Learn in specific terms what decisions are going to be made on 
the basis of your information. 

3. Outline the facts that will be needed to make those decisions. 

4. Identify respondents who have the information you need/ 

5. Use a procedure that will provide information of sufficient 
reliability. 

B. The Question Type 

1. Open-ended questions 

a. Before using an open-ended question, see whether it can be 
replaced with a multiple-choice question. (Sometimes it 
cannot.) 

b. Before using an open-ended question, be certain that you 
will have enough time to analyze the data. 
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Make a particularly strong effort to avoid open-ended ques- 
tions when the respondent does not have education beyond 
the eighth-grade level. (Some would have said hi^ school 
level.) 



Two-way questions 

a. Be certain that two alternatives are enough. 

b Try to use two forms of the question, in two different 
questionnaires, with the order of appearance of the altema- 
Uves reversed from one to the other. (This wUl counter any 
tendency people may have to choose the first or last 
answer they see.) 

c. Be certain that the alternatives do not overlap. 

d. Be certain that you will know what "was really meant" by 
each answer. 

Multiple-choice questions 

a. Check to be certain that you have thought of all reasonable 
alternatives. (Cross check this during pretesting.) 

b If you are asking people to choose from a list of nunibers, 
be certain that **response ^t" tendencies to choose from 
the middle of the list wUl not bias responses to your 
question. (Consider using different forms of the question, 
with alternatives in different order, for different parts of 
your sample.) 

c Similarly, if you have alternatives that are lengthy, make 
sure the tendency to choose from the beginning or end of 
the list dras not bias your answers. (Again, consider using 
different forms, and keep the number of words m each 
r^aponse as low as possible.) 

d. Be certain the item does not include an alternative that wttl 
overwhelm the other alternatives. 
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Writing Questions 

1. Initial development 

a. Don't use terms or ask questions about topics that your 
respondent will not know about., 

b. Make sure that any answer the respondent makes will have 
a clear meaning to you. 

c. Don*t ask people to make undesirable choices. 

d. Don't lead your respondents by wording the item ^ they 
will know what you want. 

e. Make the item and the choices short, and use simple words. 

f. Make certain the item clearly specifies what you want the 
respondent to do. 

g. Don't ask respondents to do several things in one item. 

h. Don't ask for several kinds of information in one item. 

i. Don't use tricky wording, especially double negatives. 

j. Don't ask for Hne distinctions, unless you know they will 
be meaningful to your .espondents. 

k. Don't word your question in such a way that the balance 
of responses will unavokiably be in one direction (loading). 

1. Be sure your answers are worded so that someone who 
disagrees with you can find an answer presenting his side of 
the issue. 

m. Screen your items for "All-American" words or "dead give- 
away" words. 

2. Pretesting 

a. For pretesting, use subjects representative of your eventual 
tai^et. 
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b. Ask your respondent to provide both an answor and a 
reason for the answer. 

c. Ask wheth/er there are other answers that couW have been 
given. (' 

3. One last look ^ 

a. Re-read each equestion» and apply the "so what" criterion. 
(Will this information really make a difference to the 
decision maker?) 

b. Re-examine each of the elements of information originally 
identified as necessary* and be sure that there is at least 
one- question on each, and that the responses wUl provide a 
clear basis for that part of the decision. 

c. Read the questionnaire as a whole to check that: 

(1) The flow of items as a whole will not confuse the 
respondent. 

(2) The flow of items as a whole will not tip him off as 
to what answer you think is correct. 

Using this check list and the thoughts contained in the preceding 
sections wUl not guarantee a successful questionnaire or a successful test. 
Hrwever. it is hoped that they will make the job a little faster, or a little 
easier, and a little better. 
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